Image processing device, solid-state imaging device, and camera module

ABSTRACT

According to one embodiment, an image processing device includes a shading correcting unit, a distortion correcting unit, a lens-characteristic estimating unit, and a resolution restoring unit. The shading correcting unit, the distortion correcting unit, the lens-characteristic estimating unit, the resolution restoring unit carry out signal processing for each of image data obtained by a plurality of sub-camera modules.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromthe prior Japanese Patent Application No. 2009-190147, filed on Aug. 19,2009; the entire contents of all of which are incorporated herein byreference.

FIELD

Embodiments described herein relate generally to an image processingdevice, a solid-state imaging device, and a camera module.

BACKGROUND

In a camera module used in a digital camera or the like, according to ademand for a reduction in thickness and a reduction in size, a distancebetween a lens and an imaging element (a focal length) tends to bereduced as much as possible. In the past, to reduce the distance betweenthe lens and the imaging element, a reduction in size of pixels isadvanced. A reduction in the focal length of the lens is performed by,for example, increasing an angle of the lens or using a highrefractive-index material. However, as the pixels are further reduced insize, an SN ratio is deteriorated because of insufficiency ofsensitivity due to a decrease in a light reception amount per one pixelor because of a decrease in the number of saturated electrons. Further,there is a limit in a reduction in the focal length through selection ofa material of the lens and design of the lens.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic perspective view of a camera module according to afirst embodiment;

FIG. 2 is a schematic top view of an image sensor unit;

FIG. 3 is a block diagram of a configuration for signal processing inthe camera module;

FIG. 4 is a diagram for explaining alignment of subject images by ablock matching unit;

FIG. 5 is a block diagram of a modification of the configuration for thesignal processing in the camera module;

FIG. 6 is a schematic perspective view of a camera module according to asecond embodiment;

FIG. 7 is a schematic top view of an image sensor unit;

FIG. 8 is a block diagram of a configuration for signal processing inthe camera module;

FIGS. 9A to 9C are diagrams for explaining parallax of subject images;and

FIG. 10 is a block diagram of a modification of the configuration forthe signal processing in the camera module.

DETAILED DESCRIPTION

In general, according to one embodiment, an image processing deviceincludes a shading correcting unit, a distortion correcting unit, alens-characteristic estimating unit, a resolution restoring unit, ablock matching unit, and a demosaicing unit. The shading correcting unitcarries out shading correction for subject images picked up by imagingelements. The distortion correcting unit corrects distortion of thesubject images. The lens-characteristic estimating unit estimates lenscharacteristics of imaging lenses that capture light made incident onthe imaging elements. The resolution restoring unit carries out, basedon the estimated lens characteristics, resolution restoration processingfor the subject images. The block matching unit carries out blockmatching processing for alignment of the subject images. The demosaicingunit generates a color image by performing demosaicing processing forimage data obtained by the block matching processing. The shadingcorrecting unit, the distortion correcting unit, the lens-characteristicestimating unit, and the resolution restoring unit carry out, on aplurality of sub-camera modules including the imaging elements and theimaging lenses, signal processing for each of image data obtained by thesub-camera modules.

Exemplary embodiments of an image processing device, a solid-stateimaging device, and a camera module will be explained below in detailwith reference to the accompanying drawings. The present invention isnot limited to the following embodiments.

FIG. 1 is a schematic perspective view of a camera module 10 accordingto a first embodiment. The camera module 10 includes an image sensorunit 11 and a lenslet 12. The image sensor unit 11 is a solid-stateimaging device for picking up a subject image and includes four imagingelements 13. The lenslet 12 includes four imaging lenses 14 arranged ona plane to correspond to the imaging elements 13. A housing 19 of thecamera module 10 houses the image sensor unit 11 and the lenslet 12.

The camera module 10 includes four independent sub-camera modulesincluding the imaging elements 13 and the imaging lenses 14. The imaginglenses 14 capture light from a subject and make the light incident onthe imaging elements 13. The imaging elements 13 convert the lightcaptured by the imaging lenses 14 into signal charges and pick upsubject images.

FIG. 2 is a schematic top view of the image sensor unit 11. The fourimaging elements 13 (13Gr, 13R, 13B, and 13Gb) are arranged in a 2×2matrix shape. The imaging element 13R for red (R) light, the imagingelement 13B for blue (B) light, and the two imaging elements 13Gr and13Gb for green (G) light are arranged such that the two imaging elements13Gr and 13Gb for G light are diagonally opposite to each other as inthe Bayer array. A sub-camera module for R includes the imaging element13R that picks up an R component of a subject image. A sub-camera modulefor B includes the imaging element 13B that picks up a B component ofthe subject image. A sub-camera module for Gr includes the imagingelement 13Gr that picks up a G component of the subject image. Asub-camera module for Gb includes the imaging element 13Gb that picks upthe G component of the subject image.

The camera module 10 adopts a configuration including the lenslet 12 tothereby reduce a focal length of the imaging lens 14. This makes itpossible to reduce a distance between the imaging elements 13 and theimaging lenses 14. In the camera module 10, because pixels for the samecolor components are included in the sub-camera modules, interference ofsignals for different color components among the adjacent pixels can beprevented. This makes it possible to reduce color mixture andsubstantially improve sensitivity. In the imaging lenses 14 of thesub-camera modules, lens design can be optimized for the respectivecolor components. This makes it possible to substantially reducelongitudinal chromatic aberration.

The sub-camera module for Gr among the four sub-camera modules is set asa reference sub-camera module. In a plane shown in FIG. 2, a directionin which the imaging element 13Gr of the reference sub-camera module andthe imaging element 13R of the sub-camera module for R are arranged inparallel is represented as X direction. A direction in which the imagingelement 13Gr of the reference sub-camera module and the imaging element13B of the sub-camera module for B are arranged in parallel isrepresented as Y direction. The X direction and the Y direction areperpendicular to each other.

Intersections of broken lines shown in the figure indicate centerpositions of the imaging elements 13Gr, 13R, 13B, and 13Gb when it isassumed that focusing positions of subject images of the colorcomponents coincide with one another. When a focusing position of asubject image picked up by the sub-camera module for Gr is set as areference, the imaging element 13R of the sub-camera module for R isarranged such that a focusing position of a subject image thereof isshifted by a half pixel in the X direction with respect to thereference. The imaging element 13B of the sub-camera module for B isarranged such that a focusing position of a subject image thereof isshifted by a half pixel in the Y direction with respect to thereference. The imaging element 13Gb of the sub-camera module for Gb isarranged such that a focusing position of a subject image thereof isshifted by a half pixel in the X direction and the Y direction withrespect to the reference. Because the sub-camera module for G light isset as the reference sub-camera module and the subject image of the Gcomponent having high luminous efficiency is set as the reference, anaccuracy difference in image processing explained later can be reduced.

FIG. 3 is a block diagram of a configuration for signal processing inthe camera module 10. The configuration for the signal processing isroughly divided into the image sensor unit 11 at a pre-stage and adigital signal processor (DSP) 20 at a post-stage. The DSP 20 is animage processing device that processes image data from the image sensorunit 11. The image sensor unit 11 includes a shading correcting unit 15,a distortion correcting unit 16, a lens-characteristic estimating unit17, and a resolution restoring unit 18. The shading correcting unit 15,the distortion correcting unit 16, the lens-characteristic estimatingunit 17, and the resolution restoring unit 18 carry out the signalprocessing for each of RAW images of Gr, R, G, and Gb obtained by thefour sub-camera modules.

The shading correcting unit 15 corrects luminance unevenness caused bythe imaging lenses 14, in particular, a light amount difference betweenthe center and the periphery of a subject image (shading correction).The distortion correcting unit 16 corrects distortion of the subjectimage caused by the imaging lenses 14. The lens-characteristicestimating unit 17 estimates lens characteristics of the imaging lenses14 such as magnification chromatic aberration and a blur amount that arecauses of color blurring of a contour. As the lens characteristics, forexample, a point spread function (PSF) as an optical transfercoefficient is used. The lens-characteristic estimating unit 17estimates the PSF according to, for example, the method of leastsquares.

The resolution restoring unit 18 carries out resolution restorationprocessing based on the lens characteristics estimated by thelens-characteristic estimating unit 17. An effect of resolutionrestoration depends on an algorithm used for restoration. In theresolution restoration processing, for example, the Richardson-Lucymethod is used to restore an image close to an original subject image.

The DSP 20 includes a block matching unit 21 and a demosaicing unit 22.The block matching unit 21 carries out block matching (pattern matching)processing on the RAW images of Gr, R, B, and Gb subjected to theprocessing by the shading correcting unit 15, the distortion correctingunit 16, the lens-characteristic estimating unit 17, and the resolutionrestoring unit 18. The block matching unit 21 aligns, by performing theblock matching processing, subject images obtained by the sub-cameramodules.

FIG. 4 is a diagram for explaining the alignment of the subject imagesby the block matching unit 21. All squares shown in the figure representpixels. Concerning an R pixel, a B pixel, and a Gb pixel, a state inwhich focusing positions of subject images coincide with one another isrepresented by a broken line and a state in which the pixels are shiftedwith respect to a Gr pixel is represented by a solid line. The R pixelis shifted by a half pixel in a lateral direction in the figure withrespect to the Gr pixel. The B pixel is shifted by a half pixel in alongitudinal direction in the figure with respect to the Gr pixel. TheGb pixel is shifted by a half pixel in the lateral direction and thelongitudinal direction with respect to the Gr pixel. The block matchingunit 21 performs, based on the position of the Gr pixel, alignment insub-pixel units such that the R pixel, the B pixel, and the Gb pixel areshifted by a half pixel in predetermined directions. Consequently, theblock matching unit 21 carries out block matching processing forobtaining a predetermined total number of pixels.

Referring back to FIG. 3, the demosaicing unit 22 generates a colorimage by performing demosaicing processing for an image obtained by theblock matching processing. The demosaicing unit 22 generates a signalvalue of an insufficient color component by applying pixel interpolationprocessing to the image obtained by the block matching processingassuming that the image is formed in the Bayer array. In thisembodiment, the subject images picked up by the sub-camera modules areshifted to generate a color image, whereby the predetermined totalnumber of pixels is obtained. The camera module 10 outputs the colorimage generated in this way. The procedure of the processing explainedin this embodiment is only an example. Addition of other processing, achange of order of the processing, and the like can be performed asappropriate.

The camera module 10 according to this embodiment can performphotographing at high sensitivity by reducing color mixture and carryingout the signal processing for each of the RAW images. Consequently, thecamera module 10 can realize a reduction in thickness and a reduction insize and the photographing at high sensitivity. The camera module 10 isconfigured to carry out RAW image processing in the image sensor unit 11and carry out RGB synchronization (demosaicing) processing in the DSP20. Therefore, most of processing by the camera module in the past canbe diverted and changes of a system can be reduced. In the camera module10, the system in the past can be diverted concerning, for example,processing other than the block matching processing.

In the camera module 10, sub-camera modules including, for example,low-sensitivity pixels and high-sensitivity pixels for G light, pixelsincluding complementary color filters for colors other than RGB, andpixels including white/gray filters can be provided instead of thesub-camera modules for Gr and Gb according to a purpose or anapplication.

The camera module 10 does not always shift subject images according tothe arrangement of the sub-camera modules. The camera module 10 canshift the subject images by, for example, adding position informationconcerning the shift of the subject images to the lens characteristicsestimated by the lens-characteristic estimating unit 17. Positioninformation for shifting the subject images picked up by the sub-cameramodules other than the reference sub-camera module with respect to thesubject images picked up by the reference sub-camera module is added tothe lens characteristics. The resolution restoring unit 18 restoresresolution based on the lens characteristics added with such positioninformation. In this way, the camera module 10 shifts the subject imagesby performing the signal processing. Such a method is useful, forexample, when it is difficult to physically control a shift amount of asubject because influence of an attachment error of an imaging elementor fluctuation in manufacturing is large. The method is suitable forreducing the size of the imaging elements.

The camera module 10 is not limited to a camera module that obtains thepredetermined total number of pixels by shifting subject images insub-pixel units. For example, the camera module 10 can obtain thepredetermined total number of pixels through up-sampling in thedemosaicing unit 22. The up-sampling is useful when alignment insub-pixel units is difficult in the block matching unit 21. When theup-sampling in the demosaicing unit 22 is adopted, the camera module 10can also obtain high sensitivity.

FIG. 5 is a block diagram of a modification of a configuration for thesignal processing in the camera module 10. In this modification, thecamera module 10 carries out the signal processing from the shadingcorrection to the demosaicing processing in the DSP 20. The DSP 20includes the shading correcting unit 15, the distortion correcting unit16, the lens-characteristic estimating unit 17, the resolution restoringunit 18, the block matching unit 21, and the demosaicing unit 22. Theimage sensor unit 11 includes a parameter storing unit 23. Parametersnecessary for the processing in the DSP 20 are written in the parameterstoring unit 23. The parameter storing unit 23 stores the writtenparameters. The image sensor unit 11 stores individual information ofthe camera module 10 in the parameter storing unit 23 as parameters. Theindividual information is information concerning an individualdifference of each product, for example, a manufacturing error of acomponent such as a lens and an assembly error among components.

The shading correcting unit 15 carries out the shading correction for asubject image referring to the parameters stored in the parameterstoring unit 23. The distortion correcting unit 16 corrects distortionof the subject image referring to the parameters stored in the parameterstoring unit 23. The lens-characteristic estimating unit 17 estimatesthe lens characteristics of the imaging lenses 14 of the sub-cameramodules referring to the parameters stored in the parameter storing unit23. In the camera module 10, at least one of the shading correcting unit15, the distortion correcting unit 16, and the lens-characteristicestimating unit 17 only has to refer to the parameters stored in theparameter storing unit 23.

In the case of this modification, as in the embodiment, the cameramodule 10 can realize a reduction in thickness and a reduction in sizeand the photographing at high sensitivity. In general, in the DSP 20,limitation on a circuit size is often small compared with the imagesensor unit 11. Therefore, the camera module 10 is configured to carryout the RAW image processing and the RGB synchronization (demosaicing)processing in the DSP 20. This makes it possible to obtain ahigh-quality image by performing complicated and advanced signalprocessing. Further, in the case of this modification, in the cameramodule 10, the circuit size of the image sensor unit 11 can be reduced.

The camera module 10 can carry out, with the image sensor unit 11, thesignal processing from the shading correction to the demosaicingprocessing. In this case, the image sensor unit 11 includes the shadingcorrecting unit 15, the distortion correcting unit 16, thelens-characteristic estimating unit 17, the resolution restoring unit18, the block matching unit 21, and the demosaicing unit 22 (not shownin the figure). The camera module 10 is configured to carry out the RAWimage processing and the RGB synchronization (demosaicing) processing inthe image sensor unit 11. This makes it possible to increase speed ofthe signal processing with a simple circuit configuration. The DSP 20 isnot always provided on the inside of the camera module 10 and can beprovided on the outside of the camera module 10.

FIG. 6 is a schematic perspective view of a camera module 30 accordingto a second embodiment. The camera module 30 according to thisembodiment includes nine independent sub-camera modules includingimaging elements 33 and imaging lenses 34. An image sensor unit 31 as asolid-state imaging device includes nine imaging elements 33. A lenslet32 includes nine imaging lenses 34 arranged on a plane to correspond tothe imaging elements 33. A housing 19 of the camera module 30 houses theimage sensor unit 31 and the lenslet 32.

FIG. 7 is a schematic top view of the image sensor unit 31. The nineimaging elements 33 are arranged in a 3×3 matrix shape. Among the nineimaging elements 33, two imaging elements 33 are imaging elements for Rlight (R01 and R12), two imaging elements 33 are imaging elements for Blight (B10 and B21), and five imaging elements are imaging elements forG light (G00, G02, G11, G20, and G22). One imaging element 33 (G11)among the five imaging elements 33 for G light is arranged in the centerof the matrix. Four imaging elements 33 (G00, G02, G20, and G22) arearranged to be diagonally opposed around the imaging element G11. Theimaging elements 33 for R light (R01 and R12) and the imaging elements33 for B light (B10 and B21) are arranged to be longitudinally andlaterally adjacent to the imaging elements 33 for G light.

Like the camera module 10 according to the first embodiment, in thecamera module 30 according to this embodiment, a reduction in a distancebetween the imaging elements 33 and the imaging lenses 34, improvementof sensitivity through a reduction in color mixture, and a reduction inlongitudinal chromatic aberration are possible. A sub-camera module forG (G11) arranged in the center among the nine sub-camera modules is setas a reference sub-camera module. Because the sub-camera module for Glight is set as the reference sub-camera module and a subject image of aG component having high luminous efficiency is set as a reference, anaccuracy difference in image processing explained later can be reduced.

FIG. 8 is a block diagram of a configuration for signal processing inthe camera module 30. The configuration for the signal processing isroughly divided into the image sensor unit 31 at a pre-stage and a DSP40 at a post-stage. The DSP 40 is an image processing device thatprocesses image data from the image sensor unit 31. The image sensorunit 31 includes a shading correcting unit 35, a distortion correctingunit 36, a lens-characteristic estimating unit 37, and a resolutionrestoring unit 38. In the image sensor unit 31, the shading correctingunit 35, the distortion correcting unit 36, the lens-characteristicestimating unit 37, and the resolution restoring unit 38 carry out thesignal processing for each of RAW images obtained by the nine sub-cameramodules (G00, R01, G02, B10, G11, R12, G20, B21, and G22).

The shading correcting unit 35 corrects luminance unevenness caused bythe imaging lenses 34, in particular, a light amount difference betweenthe center and the periphery of a subject image (shading correction).The distortion correcting unit 36 corrects distortion of the subjectimage caused by the imaging lenses 34. The lens-characteristicestimating unit 37 estimates lens characteristics of the imaging lenses34 such as magnification chromatic aberration and a blur amount that arecauses of color blurring of a contour. As the lens characteristics, forexample, a point spread function (PSF) as an optical transfercoefficient is used. The lens-characteristic estimating unit 37estimates the PSF according to, for example, the method of leastsquares.

The resolution restoring unit 38 carries out resolution restorationprocessing based on the lens characteristics estimated by thelens-characteristic estimating unit 37. An effect of resolutionrestoration depends on an algorithm used for restoration. In theresolution restoration processing, for example, the Richardson-Lucymethod is used to restore an image close to an original subject image.

The DSP 40 includes a block matching unit 41, a weighting processingunit 42, a sampling unit 43, and a demosaicing unit 44. The blockmatching unit 41 carries out block matching (pattern matching)processing on the RAW images subjected to the processing by the shadingcorrecting unit 35, the distortion correcting unit 36, thelens-characteristic estimating unit 37, and the resolution restoringunit 38. The block matching unit 41 aligns, by performing the blockmatching processing, subject images obtained by the sub-camera modules.The weighting processing unit 42 calculates a parallax amount in thesubject images obtained by the sub-camera modules and carries outweighting processing corresponding to the parallax amount to therebycorrect parallax.

FIGS. 9A to 9C are diagrams for explaining parallax of a subject image.In FIG. 9A, a subject P1 is present at the infinity. In this case, noparallax occurs in the subject images obtained by the sub-cameramodules. In FIG. 9B, a subject P2 is present at a close-range distance.Parallax is a phenomenon in which focusing positions are different atthe close-range distance. When parallax occurs, if a color image isgenerated without taking into account the parallax, an image is blurredand image quality is spoiled. Subject images obtained by the eightsub-camera modules around the reference sub-camera module among the ninesub-camera modules are affected by the parallax.

On the other hand, in the case of the reference camera module located inthe center of the nine sub-camera modules, as shown in FIG. 9C, when thesubject P1 is present at the infinity and the subject P2 is present atthe close-range distance, assuming that parallax does not occur in thesubject images, the weighting processing unit 42 calculates a parallaxamount based on difference information between a subject image picked upby the reference sub-camera module not affected by parallax and subjectimages picked up by the other sub-camera modules. As the differenceinformation, for example, a luminance difference and edge informationare applied. The weighting processing unit 42 calculates differenceinformation in sub-pixel units between the diagonally-arrangedsub-camera modules (G00, G02, G20, and G22) shown in FIG. 7 and thereference sub-camera module (G11). Output resolution is higher as thedifference information is calculated at a finer sub-pixel level. Theweighting processing unit 42 increases weighting on a signal value ofthe reference sub-camera module (G11) assuming that a parallax amount islarger as a calculated difference value is larger. On the other hand,concerning the R light, because luminous efficiency is not as high asthat of the G light, the weighting processing unit 42 performs weightingbased on weighting information of one sub-camera module (G02) and thereference sub-camera module (G11). Concerning the B light, as in thecase of the R light, the weighting processing unit 42 performs weightingbased on weighting information of one sub-camera module (G20) and thereference sub-camera module (G11).

In the camera module 30, a coordinate conversion processing unit (notshown in the figure) that reduces a parallax amount (a movement amount)by performing coordinate conversion processing employing a matrix suchas Affine transform can be applied instead of the weighting processingunit 42. The coordinate conversion processing means a processing forcalculating, based on position information of a pixel photographed bythe reference sub-camera module and position information of pixelsphotographed by the other sub-camera modules, a coordinate conversionmatrix through a method such as the method of least squares. In thisembodiment, the nine sub-camera modules are used. However, when foursub-camera modules are used, it is desirable to calculate a parallaxamount with the imaging element for G light set as a reference.

Referring back to FIG. 8, the sampling unit 43 carries out, with asubject image picked up by the reference sub-camera module set as areference, sampling of a signal value concerning an image, parallax ofwhich is corrected, and obtains a predetermined total number of pixels.The sampling unit 43 carries out, based on an image generated insub-pixel units by the weighting processing unit 42, sampling forobtaining a signal value in pixel units. As method of sampling, forexample, an interpolation method such as a bilinear or bicubic method isapplied. When the coordinate conversion processing unit is appliedinstead of the weighting processing unit 42, the sampling unit 43carries out sampling based on an image generated by the coordinateconversion processing unit. The demosaicing unit 44 generates a colorimage by performing demosaicing processing for an image obtained by thesampling in the sampling unit 43. The camera module 30 outputs the colorimage generated in this way. A procedure of the processing explained inthis embodiment is only an example. Addition of other processing, achange of the procedure of the processing, and the like can be performedas appropriate.

In this embodiment, as in the first embodiment, the camera module 30 canrealize a reduction in thickness and a reduction in size andphotographing at high sensitivity. The camera module 30 is configured tocarry out RAW image processing in the image sensor unit 31 and carry outRGB synchronization (demosaicing) processing in the DSP 40. Therefore,most of processing by the camera module in the past can be diverted andchanges of a system can be reduced. In the camera module 30, the systemin the past can be diverted concerning, for example, processing otherthan the block matching processing.

The arrangement of the sub-camera modules is not limited to that shownin FIG. 7. For example, the arrangement of the sub-camera module for Rand the sub-camera module for B can be changed as appropriate. As thereference sub-camera module, a sub-camera module in which RGB pixels arearranged in the Bayer array can be used instead of the sub-camera modulefor G. In this case, a focal length of the sub-camera module in whichthe RGB pixels are arranged in the Bayer array is long. Therefore, inthe camera module 30, it is desirable to use wide-angle lenses as theimaging lenses 34 to equalize the focal length with a focal length ofthe other sub-camera modules. Further, in the camera module 30,sub-camera modules including, for example, low-sensitivity pixels andhigh-sensitivity pixels, pixels including complementary color filtersfor colors other than RGB, and pixels including white/gray filters canbe provided according to a purpose or an application.

The camera module 30 does not always obtain the predetermined totalnumber of pixels through the image generation in sub-pixel units in theweighting processing unit 42. Even when accuracy in sub-pixel unitscannot be obtained, for example, the camera module 30 can obtain thepredetermined total number of pixels through up-sampling in thedemosaicing unit 44. When the up-sampling in the demosaicing unit 44 isadopted, the camera module 30 can also obtain high sensitivity.

FIG. 10 is a block diagram of a modification of the configuration forthe signal processing in the camera module 30. In this modification, thecamera module 30 carries out the signal processing from the shadingcorrection to the demosaicing processing in the DSP 40. The DSP 40includes the shading correcting unit 35, the distortion correcting unit36, the lens-characteristic estimating unit 37, the resolution restoringunit 38, the block matching unit 41, the weighting processing unit 42,the sampling unit 43, and the demosaicing unit 44. The image sensor unit31 includes a parameter storing unit 45. Parameters necessary for theprocessing in the DSP 40 are written in the parameter storing unit 45.The parameter storing unit 45 stores the written parameters. The imagesensor unit 31 stores individual information of the camera module 30 inthe parameter storing unit 45 as parameters.

The shading correcting unit 35 carries out the shading correction for asubject image referring to the parameters stored in the parameterstoring unit 45. The distortion correcting unit 36 corrects distortionof the subject image referring to the parameters stored in the parameterstoring unit 45. The lens-characteristic estimating unit 37 estimatesthe lens characteristics of the imaging lenses 34 of the sub-cameramodules referring to the parameters stored in the parameter storing unit45.

In the case of this modification, as in the embodiment, the cameramodule 30 can realize a reduction in thickness and a reduction in sizeand the photographing at high sensitivity. In general, in the DSP 40,limitation on a circuit size is often small compared with the imagesensor unit 31. Therefore, the camera module 30 is configured to carryout the RAW image processing and the RGB synchronization (demosaicing)processing in the DSP 40. This makes it possible to obtain ahigh-quality image by performing complicated and advanced signalprocessing. Further, in the case of this modification, in the cameramodule 30, the circuit size of the image sensor unit 31 can be reduced.In the case of this modification, as in the embodiment, a coordinateconversion processing unit (not shown in the figure) that reduces aparallax amount (a movement amount) by performing coordinate conversionprocessing employing a matrix such as Affine transform can be appliedinstead of the weighting processing unit 42.

The camera module 30 can carry out, with the image sensor unit 31, thesignal processing from the shading correction to the demosaicingprocessing. In this case, the image sensor unit 31 includes the shadingcorrecting unit 35, the distortion correcting unit 36, thelens-characteristic estimating unit 37, the resolution restoring unit38, the block matching unit 41, the weighting processing unit 42, thesampling unit 43, and the demosaicing unit 44 (not shown in the figure).The camera module 30 is configured to carry out the RAW image processingand the RGB synchronization (demosaicing) processing in the image sensorunit 31. This makes it possible to increase speed of the signalprocessing with a simple circuit configuration. In this case, as in theembodiment, a coordinate conversion processing unit (not shown in thefigure) that reduces a parallax amount (a movement amount) by performingcoordinate conversion processing employing a matrix such as Affinetransform can be applied instead of the weighting processing unit 42.The DSP 40 is not always provided on the inside of the camera module 30and can be provided on the outside of the camera module 30.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel devices and modules describedherein may be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the devices andmodules described herein may be made without departing from the spiritof the inventions. The accompanying claims and their equivalents areintended to cover such forms or modifications as would fall within thescope and spirit of the inventions.

1. An image processing device comprising: a shading correcting unit thatcarries out shading correction for subject images picked up by imagingelements; a distortion correcting unit that corrects distortion of thesubject images; a lens-characteristic estimating unit that estimateslens characteristics of imaging lenses that capture light made incidenton the imaging elements; a resolution restoring unit that carries out,based on the estimated lens characteristics, resolution restorationprocessing for the subject images; a block matching unit that carriesout block matching processing for alignment of the subject images; and ademosaicing unit that generates a color image by performing demosaicingprocessing for image data obtained by the block matching processing,wherein the shading correcting unit, the distortion correcting unit, thelens-characteristic estimating unit, and the resolution restoring unitcarry out, on a plurality of sub-camera modules including the imagingelements and the imaging lenses, signal processing for each of imagedata obtained by the sub-camera modules.
 2. The image processing deviceaccording to claim 1, wherein one of the sub-camera modules is set as areference sub-camera module, and the shading correcting unit, thedistortion correcting unit, the lens-characteristic estimating unit, andthe resolution restoring unit carry out the signal processing on theimage data obtained by shifting focusing positions of the subject imagesby the sub-camera modules other than the reference sub-camera modulewith respect to a focusing position of the subject image by thereference sub-camera module.
 3. The image processing device according toclaim 2, wherein the reference sub-camera module is a sub-camera modulefor green that picks up a green component of the subject image.
 4. Theimage processing device according to claim 1, wherein one of thesub-camera modules is set as a reference sub-camera module, and theresolution restoring unit carries out the resolution restorationprocessing based on the lens characteristics added with positioninformation for shifting the subject images picked up by the sub-cameramodules other than the reference sub-camera module with respect to thesubject image picked up by the reference sub-camera module.
 5. The imageprocessing device according to claim 1, wherein one of the sub-cameramodules is set as a reference sub-camera module, and the imageprocessing device further comprises a sampling unit that carries out,with the subject image picked up by the reference sub-camera module setas a reference, sampling of a signal value concerning an image, parallaxof which due to the sub-camera modules other than the referencesub-camera module is corrected.
 6. The image processing device accordingto claim 5, further comprising a weighting processing unit thatcalculates a parallax amount of the subject images obtained by thesub-camera modules and carries out weighting processing corresponding tothe parallax amount, wherein the sampling unit carries out the samplingbased on an image generated by the weighting processing unit.
 7. Theimage processing device according to claim 5, further comprising acoordinate conversion processing unit that carries out coordinateconversion processing for reducing a parallax amount of the subjectimages obtained by the sub-camera modules, wherein the sampling unitcarries out the sampling based on an image generated by the coordinateconversion processing unit.
 8. The image processing device according toclaim 1, wherein the block matching unit carries out the block matchingprocessing for obtaining a predetermined total number of pixels.
 9. Theimage processing device according to claim 1, wherein the demosaicingunit carries out up-sampling for obtaining a predetermined total numberof pixels.
 10. A solid-state imaging device comprising a plurality ofimaging elements that pick up subject images, wherein one of a pluralityof sub-camera modules including the imaging elements and imaging lenses,which capture light made incident on the imaging elements, is set as areference sub-camera module, and the imaging elements of the sub-cameramodules other than the reference sub-camera module are arranged withfocusing positions of the subject images shifted with respect to theimaging element of the reference sub-camera module.
 11. The solid-stateimaging device according to claim 10, wherein the imaging element of thereference sub-camera module is arranged in a center of the imagingelements.
 12. The solid-state imaging device according to claim 10,further comprising: a shading correcting unit that carries out shadingcorrection for the subject images; a distortion correcting unit thatcorrects distortion of the subject images; a lens-characteristicestimating unit that estimates lens characteristics of the imaginglenses that capture light made incident on the imaging elements; and aresolution restoring unit that carries out, based on the estimated lenscharacteristics, resolution restoration processing for the subjectimages, wherein the shading correcting unit, the distortion correctingunit, the lens-characteristic estimating unit, and the resolutionrestoring unit carry out signal processing for each of image dataobtained by the sub-camera modules.
 13. The solid-state imaging deviceaccording to claim 12, further comprising: a block matching unit thatcarries out, on image data subjected to the processing by the shadingcorrecting unit, the distortion correcting unit, the lens-characteristicestimating unit, and the resolution restoring unit, block matchingprocessing for alignment of the subject images; and a demosaicing unitthat generates a color image by performing demosaicing processing for animage obtained by the block matching processing.
 14. A camera modulecomprising: a plurality of imaging elements that pick up subject images;a plurality of imaging lenses that capture light made incident on theimaging elements; a housing that houses the imaging elements and theimaging lenses; a shading correcting unit that carries out shadingcorrection for the subject images; a distortion correcting unit thatcorrects distortion of the subject images; a lens-characteristicestimating unit that estimates lens characteristics of the imaginglenses; a resolution restoring unit that carries out, based on theestimated lens characteristics, resolution restoration processing forthe subject images; a block matching unit that carries out blockmatching processing for alignment of the subject images; and ademosaicing unit that generates a color image by performing demosaicingprocessing for an image obtained by the block matching processing,wherein the shading correcting unit, the distortion correcting unit, thelens-characteristic estimating unit, and the resolution restoring unitcarry out, on a plurality of sub-camera modules including the imagingelements and the imaging lenses, signal processing for each of imagedata obtained by the sub-camera modules.
 15. The camera module accordingto claim 14, wherein one of the sub-camera modules is set as a referencesub-camera module, and the imaging elements of the sub-camera modulesother than the reference sub-camera module are arranged with focusingpositions of the subject images shifted with respect to the imagingelement of the reference sub-camera module.
 16. The camera moduleaccording to claim 14, wherein one of the sub-camera modules is set as areference sub-camera module, and the resolution restoring unit carriesout the resolution restoration processing based on the lenscharacteristics added with position information for shifting the subjectimages picked up by the sub-camera modules other than the referencesub-camera module with respect to the subject image picked up by thereference sub-camera module.
 17. The camera module according to claim14, wherein one of the sub-camera modules is set as a referencesub-camera module, and the camera module further comprises a samplingunit that carries out, with the subject image picked up by the referencesub-camera module set as a reference, sampling of a signal valueconcerning an image, parallax of which due to the sub-camera modulesother than the reference sub-camera module is corrected.
 18. The cameramodule according to claim 17, further comprising a weighting processingunit that calculates a parallax amount of the subject images obtained bythe sub-camera modules and carries out weighting processingcorresponding to the parallax amount, wherein the sampling unit carriesout the sampling based on an image generated by the weighting processingunit.
 19. The camera module according to claim 17, further comprising acoordinate conversion processing unit that carries out coordinateconversion processing for reducing a parallax amount of the subjectimages obtained by the sub-camera modules, wherein the sampling unitcarries out the sampling based on an image generated by the coordinateconversion processing unit.
 20. The camera module according to claim 14,further comprising a parameter storing unit that stores individualinformation of the camera module as parameters, wherein at least one ofthe shading correcting unit, the distortion correcting unit, and thelens-characteristic estimating unit refers to the parameters stored inthe parameter storing unit.